Visual indication of audibility

ABSTRACT

Various implementations disclosed herein include devices, systems, and methods for displaying a visual indication of audibility of an audible signal. In various implementations, a device includes a display, an audio sensor, a processor and a non-transitory memory. In various implementations, a method includes receiving, via the audio sensor, an audible signal and converting the audible signal to electronic signal data. In various implementations, the method includes obtaining environmental data that indicates audio response characteristics of a physical environment in which the device is located. In various implementations, the method includes displaying, on the display, an indicator that indicates a distance from a source of the audible signal at which the audible signal is audible. In some implementations, the distance is based on an amplitude associated with the electronic signal data and the audio response characteristics of the physical environment.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional Patent App. No.63/072,198, filed on Aug. 30, 2020, which is incorporated by referencein its entirety.

TECHNICAL FIELD

The present disclosure generally relates to displaying a visualindication of audibility of an audible signal.

BACKGROUND

Some devices are capable of generating and presenting extended reality(XR) environments. Some XR environments include virtual environmentsthat are simulated replacements of physical environments. Some XRenvironments include augmented environments that are modified versionsof physical environments. Some devices that present XR environmentsinclude mobile communication devices such as smartphones, tablets,head-mountable displays (HMDs), eyeglasses, heads-up displays (HUDs),and optical projection systems.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the present disclosure can be understood by those of ordinaryskill in the art, a more detailed description may be had by reference toaspects of some illustrative implementations, some of which are shown inthe accompanying drawings.

FIGS. 1A-1E are diagrams of an example operating environment inaccordance with some implementations.

FIG. 2 is a block diagram of a system in accordance with someimplementations.

FIG. 3 is a flowchart representation of a method of displaying a visualindication of audibility in accordance with some implementations.

FIG. 4A is a block diagram of a device that displays a visual indicationof audibility in accordance with some implementations.

FIG. 4B is a blow-up view of an optical see-through display inaccordance with some implementations.

In accordance with common practice the various features illustrated inthe drawings may not be drawn to scale. Accordingly, the dimensions ofthe various features may be arbitrarily expanded or reduced for clarity.In addition, some of the drawings may not depict all of the componentsof a given system, method or device. Finally, like reference numeralsmay be used to denote like features throughout the specification andfigures.

SUMMARY

Various implementations disclosed herein include devices, systems, andmethods for displaying a visual indication of audibility of an audiblesignal. In various implementations, a device includes a display, anaudio sensor, a processor and a non-transitory memory. In variousimplementations, a method includes receiving, via the audio sensor, anaudible signal and converting the audible signal to electronic signaldata. In various implementations, the method includes obtainingenvironmental data that indicates audio response characteristics of aphysical environment in which the device is located. In variousimplementations, the method includes displaying, on the display, anindicator that indicates a distance from a source of the audible signalat which the audible signal is audible. In some implementations, thedistance is based on an amplitude associated with the electronic signaldata and the audio response characteristics of the physical environment.

In accordance with some implementations, a device includes one or moreprocessors, a non-transitory memory, and one or more programs. In someimplementations, the one or more programs are stored in thenon-transitory memory and are executed by the one or more processors. Insome implementations, the one or more programs include instructions forperforming or causing performance of any of the methods describedherein. In accordance with some implementations, a non-transitorycomputer readable storage medium has stored therein instructions that,when executed by one or more processors of a device, cause the device toperform or cause performance of any of the methods described herein. Inaccordance with some implementations, a device includes one or moreprocessors, a non-transitory memory, and means for performing or causingperformance of any of the methods described herein.

DESCRIPTION

Numerous details are described in order to provide a thoroughunderstanding of the example implementations shown in the drawings.However, the drawings merely show some example aspects of the presentdisclosure and are therefore not to be considered limiting. Those ofordinary skill in the art will appreciate that other effective aspectsand/or variants do not include all of the specific details describedherein. Moreover, well-known systems, methods, components, devices, andcircuits have not been described in exhaustive detail so as not toobscure more pertinent aspects of the example implementations describedherein.

A physical environment refers to a physical world that people can senseand/or interact with without aid of electronic devices. The physicalenvironment may include physical features such as a physical surface ora physical object. For example, the physical environment corresponds toa physical park that includes physical trees, physical buildings, andphysical people. People can directly sense and/or interact with thephysical environment such as through sight, touch, hearing, taste, andsmell. In contrast, an extended reality (XR) environment refers to awholly or partially simulated environment that people sense and/orinteract with via an electronic device. For example, the XR environmentmay include augmented reality (AR) content, mixed reality (MR) content,virtual reality (VR) content, and/or the like. With an XR system, asubset of a person's physical motions, or representations thereof, aretracked, and, in response, one or more characteristics of one or morevirtual objects simulated in the XR environment are adjusted in a mannerthat comports with at least one law of physics. As one example, the XRsystem may detect head movement and, in response, adjust graphicalcontent and an acoustic field presented to the person in a mannersimilar to how such views and sounds would change in a physicalenvironment. As another example, the XR system may detect movement ofthe electronic device presenting the XR environment (e.g., a mobilephone, a tablet, a laptop, or the like) and, in response, adjustgraphical content and an acoustic field presented to the person in amanner similar to how such views and sounds would change in a physicalenvironment. In some situations (e.g., for accessibility reasons), theXR system may adjust characteristic(s) of graphical content in the XRenvironment in response to representations of physical motions (e.g.,vocal commands).

There are many different types of electronic systems that enable aperson to sense and/or interact with various XR environments. Examplesinclude head mountable systems, projection-based systems, heads-updisplays (HUDs), vehicle windshields having integrated displaycapability, windows having integrated display capability, displaysformed as lenses designed to be placed on a person's eyes (e.g., similarto contact lenses), headphones/earphones, speaker arrays, input systems(e.g., wearable or handheld controllers with or without hapticfeedback), smartphones, tablets, and desktop/laptop computers. A headmountable system may have one or more speaker(s) and an integratedopaque display. Alternatively, a head mountable system may be configuredto accept an external opaque display (e.g., a smartphone). The headmountable system may incorporate one or more imaging sensors to captureimages or video of the physical environment, and/or one or moremicrophones to capture audio of the physical environment. Rather than anopaque display, a head mountable system may have a transparent ortranslucent display. The transparent or translucent display may have amedium through which light representative of images is directed to aperson's eyes. The display may utilize digital light projection, OLEDs,LEDs, uLEDs, liquid crystal on silicon, laser scanning light source, orany combination of these technologies. The medium may be an opticalwaveguide, a hologram medium, an optical combiner, an optical reflector,or any combination thereof. In some implementations, the transparent ortranslucent display may be configured to become opaque selectively.Projection-based systems may employ retinal projection technology thatprojects graphical images onto a person's retina. Projection systemsalso may be configured to project virtual objects into the physicalenvironment, for example, as a hologram or on a physical surface.

Sometimes a user of a device is not aware of how far the user's voice iscarrying in a physical environment surrounding the device. For example,if the user is having a phone conversation, the user may not be awarethat the user is talking too loudly and disturbing other people in thephysical environment. Similarly, if the user is having an in-personconversation with another person, the user may not aware of whether theuser is speaking loud enough to be heard by the other person but notloud enough to disturb other people that are not part of theconversation. For example, if the user is having an in-personconversation with a coworker in an open office setting, then the usermay not be aware of how far the user's voice is carrying and disturbingother coworkers that are not part of the conversation. In anotherexample, if the user is failing at activating a voice-activated device,the user may not realize that the user is not speaking loud enough forthe user's voice to be detected by the voice-activated device.

The present disclosure provides methods, systems, and/or devices fordisplaying a visual indication of audibility of an audible signal. Adevice displays an indicator that indicates a distance at which anaudible signal is audible. The device determines the distance based onan audio characteristic of the audible signal and audio responsecharacteristics of a physical environment in which the device islocated. The audio response characteristics indicate how an audiblesignal propagates through the physical environment. The audio responsecharacteristics can be a function of physical obstructions in thephysical environment. The audio response characteristics can be afunction of materials used in the physical environment. For example, howan audible signal propagates through the physical environment can be afunction of sound absorption characteristics or sound reflectioncharacteristics of the materials in the physical environment.

Displaying the indicator enhances a functionality of the device byallowing the user to be more aware of how far an audible signal ispropagating in the physical environment. Displaying the indicatorimproves operability of the device by allowing the user to control anamplitude of the audible signal with more certainty and precision. Forexample, if the audible signal is being generated by the device and theindicator indicates that the audible signal is audible in a particulararea of the physical environment (e.g., in a sleeping child's bedroom),the user can provide a user input to adjust an amplitude of the audiblesignal until the indicator indicates that the audible signal is notaudible in that particular area of the physical environment. Displayingthe indicator likely reduces a number of user inputs corresponding toadjusting an amplitude of the audible signal. For example, in theabsence of the indicator the user may need to alternate between going tothe child's bedroom and adjusting the amplitude of the audible signaluntil the audible signal is not audible in the child's bedroom. As such,displaying the indicator tends to reduce a number of user inputscorresponding to adjusting an amplitude of an audible signal. Reducing anumber of unnecessary user inputs tends to prolong a battery life of abattery-powered device thereby enhancing operability of the device.

FIG. 1A is a diagram of an example physical environment 10 in accordancewith some implementations. While pertinent features are shown, those ofordinary skill in the art will appreciate from the present disclosurethat various other features have not been illustrated for the sake ofbrevity and so as not to obscure more pertinent aspects of the exampleimplementations disclosed herein. To that end, as a non-limitingexample, the physical environment 10 includes a floor 12, a room divider14 (e.g., an office partition), a first office desk 16, a second officedesk 18, an electronic device 20, a user 30 of the electronic device 20,and a person 40.

In some implementations, the electronic device 20 includes a handheldcomputing device that can be held by the user 30. For example, in someimplementations, the electronic device 20 includes a smartphone, atablet, a media player, a laptop, or the like. In some implementations,the electronic device 20 includes a wearable computing device that canbe worn by the user 30. For example, in some implementations, theelectronic device 20 includes a head-mountable device (HMD) that can beworn around a head of the user 30, an electronic watch or a pair ofheadphones.

In some implementations, the electronic device 20 includes an opticalsee-through display (e.g., the optical see-through display 420 shown inFIG. 4B). For example, the electronic device 20 includes an HMD with anoptical see-through display. In various implementations, the opticalsee-through display is transparent. In some implementations, the opticalsee-through display includes an additive light field display (“additivedisplay”, hereinafter for the sake of brevity). In some implementations,the additive display includes a set of one or more optical holographicoptical elements (HOEs). In some implementations, the additive displaydisplays content by adding light and does not subtract or remove light.

As illustrated in FIG. 1A, in various implementations, the electronicdevice 20 detects an audible signal 50. In some implementations, theaudible signal 50 is characterized by a first audio characteristic value52 a (e.g., a first amplitude value). In some implementations, theaudible signal 50 is an utterance by a person. For example, in someimplementations, the audible signal 50 is an utterance by the user 30.Alternatively, in some implementations, the audible signal 50 isgenerated by a device. For example, in some implementations, the audiblesignal 50 is generated by the electronic device 20. In someimplementations, the audible signal 50 is generated by another devicesuch as a television, a speaker, a voice-activated device, etc.

Referring to FIG. 1B, in various implementations, the electronic device20 obtains environmental data 60 that indicates audio responsecharacteristics of the physical environment 10. In some implementations,the environmental data 60 indicates physical obstructions in thephysical environment 10. For example, in some implementations, theenvironmental data 60 indicates a location and physical dimensions ofthe room divider 14. In some implementations, the electronic device 20includes an image sensor (e.g., a camera), and the electronic device 20obtains the environmental data 60 by capturing an image of the physicalenvironment 10 with the image sensor. In such implementations, theelectronic device 20 performs scene analysis on the captured image toidentify the location and the physical dimensions of the physicalobstructions in the physical environment 10. In some implementations,the electronic device 20 includes a depth sensor (e.g., a depth camera),and the electronic device 20 obtains the environmental data 60 bycapturing depth data via the depth sensor. In such implementations, thedepth data indicates the location and the physical dimensions of thephysical obstructions in the physical environment 10.

In some implementations, the environmental data 60 indicates materialsthat are used in the physical environment 10. For example, in someimplementations, the environmental data 60 indicates whether the floor12 is carpeted, wooden or tiled. In some implementations, theenvironmental data 60 includes an image of the physical environment 10,and the electronic device 20 determines the materials that are used inthe physical environment 10 by performing a scene analysis on the image.

In some implementations, the environmental data 60 indicates soundabsorption characteristics or sound reflection characteristics ofvarious objects in the physical environment 10. In some implementations,the environmental data 60 indicates a sound transmission class (STC)rating or a noise reduction coefficient (NRC) of a physical object inthe physical environment 10. For example, in some implementations, theenvironmental data 60 indicates an STC rating or an NRC for the roomdivider 14. In some implementations, the environmental data 60 includesan image of the physical environment 10. In such implementations, theelectronic device 20 identifies the material composition of a physicalobject in the physical environment 10 by performing scene analysis onthe image. After identifying the material composition of the physicalobject, the electronic device 20 retrieves the STC rating or the NRC forthe material that the physical object is composed of.

In various implementations, the electronic device 20 determines adistance at which the audible signal 50 is audible in the physicalenvironment 10 based on the first audio characteristic value 52 a of theaudible signal 50 and the audio response characteristics of the physicalenvironment 10 indicated by the environmental data 60. As illustrated inFIG. 1B, in some implementations, the electronic device 20 displays anindicator 70 on a display 22 of the electronic device 20 to indicate thedistance.

Referring to FIG. 1C, in some implementations, the electronic device 20presents an XR environment 110 that corresponds to (e.g., represents)the physical environment 10 shown in FIGS. 1A and 1B. To that effect,the XR environment 110 includes an XR representation 112 of the floor 12(“XR floor 112”, hereinafter for the sake of brevity), an XRrepresentation 114 of the room divider 14 (e.g., “XR room divider 114”,hereinafter for the sake of brevity), an XR representation 116 of thefirst office desk 16 (“first XR office desk 116”, hereinafter for thesake of brevity), an XR representation 118 of the second office desk 18(“second XR office desk 118”, hereinafter for the sake of brevity), anXR representation 130 of the user 30 (“XR user 130”, hereinafter for thesake of brevity), and an XR representation 140 of the person 40 (“XRperson 140”, hereinafter for the sake of brevity).

In some implementations, the XR environment 110 is a pass-through of thephysical environment 10. For example, in some implementations, theelectronic device 20 includes an optical see-through display, and the XRenvironment 110 is an optical pass-through of the physical environment10. In some implementations, the electronic device 20 includes an opaquedisplay, and the XR environment 110 is a video pass-through of thephysical environment 10.

As illustrated in FIG. 1C, in some implementations, the XR environment110 includes an indicator 70 a that indicates a distance at which theaudible signal 50 is audible. In some implementations, the indicator 70a is a three-dimensional (3D) geometric shape. In the example of FIG.1C, the indicator 70 a is an arc. In some implementations, the indicator70 a is sphere-shaped. For example, in some implementations, the arc isa portion of a sphere (not shown). In some implementations, theindicator 70 a is in the shape of a bubble, and the audible signal 50 isaudible inside the bubble and the audible signal 50 is inaudible outsidethe bubble. As can be seen in FIG. 1C, the indicator 70 a encompassesthe XR person 140 indicating that the audible signal 50 is audible at alocation corresponding to the XR person 140. In other words, the XRperson 140 can hear the audible signal 50 because the XR person 140 ison the concave side of the arc. Displaying the indicator 70 a providesan indication to the user 30 that the person 40 can hear the audiblesignal 50. In the example of FIG. 1C, the audible signal 50 is audibleon the concave side of the arc represented by the indicator 70 a but noton the convex side of the arc represented by the indicator 70 a.

In the example of FIG. 1D, the audible signal 50 is associated with asecond audio characteristic value 52 b that is different from the firstaudio characteristic value 52 a shown in FIG. 1C. In someimplementations, the first audio characteristic value 52 a shown in FIG.1C represents a first amplitude value and the second audiocharacteristic value 52 b represents a second amplitude value that issmaller than the first amplitude value. FIG. 1D illustrates an indicator70 b (e.g., an arc, for example, a portion of a sphere) that indicateshow far the audible signal 50 is propagating in the physical environment10. The indicator 70 b indicates that, based on the second audiocharacteristic value 52 b, the audible signal 50 is propagating from theuser 30 to a location that corresponds to the indicator 70 b. As shownin FIG. 1D, the XR person 140 is on the convex side of the arcrepresented by the indicator 70 b. As such, in the example of FIG. 1D,the audible signal 50 in inaudible at a location corresponding to theperson 40. In other words, given the second audio characteristic value52 b, the person 40 cannot hear the audible signal 50.

FIG. 1E illustrates an XR environment 150 that includes an XR television152 (e.g., an XR representation of a physical television in a physicalenvironment), an XR wall 154 (e.g., an XR representation of a physicalwall in the physical environment), and an XR person 160 (e.g., an XRrepresentation of a person in the physical environment). In the exampleof FIG. 1E, the electronic device 20 displays a bubble 170 thatindicates how far an audible signal generated by the television can beheard. In various implementations, the audible signal generated by thetelevision can be heard at locations that are inside the bubble 170 butnot at locations that are outside the bubble 170. As such, the audiblesignal generated by the television cannot be heard by the personrepresented by the XR person 160.

If the user 30 wants to watch and listen to television withoutdisturbing the person represented by the XR person 160, the user 30 canuse the bubble 170 as a guide to set a volume of the television. Forexample, if the user 30 does not want the person represented by the XRperson 160 to be disturbed by the sound of the television, the user 30can set the volume of the television to a level that results in the XRperson 160 being outside the bubble 170. In the example of FIG. 1E, thebubble 170 serves as a visual aide that allows the user 30 to listen tothe television without disturbing the person represented by the XRperson 160.

In some implementations, the electronic device 20 includes an HMD thatis worn by the user 30. In some implementations, the HMD presents (e.g.,displays) an XR environment (e.g., the XR environment 110 shown in FIGS.1C and 1D, and/or the XR environment 150 shown in FIG. 1E) according tovarious implementations. In such implementations, the HMD displays theindicator 70 shown in FIG. 1B, the indicator 70 a shown in FIG. 1C, theindicator 70 b shown in FIG. 1D, and/or the bubble 170 shown in FIG. 1E.In some implementations, the HMD includes an integrated display (e.g., abuilt-in display, for example, a built-in optical see-through display ora built-in opaque display) that displays the XR environment includingthe indicator. In some implementations, the HMD includes ahead-mountable enclosure. In various implementations, the head-mountableenclosure includes an attachment region to which another device with adisplay can be attached. For example, in some implementations, anelectronic watch, a smartphone or a tablet can be attached to thehead-mountable enclosure. In various implementations, the head-mountableenclosure is shaped to form a receptacle for receiving another devicethat includes a display (e.g., an electronic watch, a smartphone or atablet). For example, in some implementations, a device with a displayslides/snaps into or otherwise attaches to the head-mountable enclosure.In some implementations, the display of the device attached to thehead-mountable enclosure presents (e.g., displays) the XR environmentincluding the indicator. In various implementations, examples of theelectronic device 20 include smartphones, tablets, media players,laptops, etc.

FIG. 2 is a block diagram of a system 200 in accordance with someimplementations. In some implementations, the system 200 resides at(e.g., is implemented by) the electronic device 20 shown in FIGS. 1A-1E.In some implementations, the electronic device 20 (shown in FIGS. 1A-1E)includes the system 200. In various implementations, the system 200includes an audio sensor 210, a data obtainer 220, a sound propagationdeterminer 230 and a content presenter 240.

In various implementations, the audio sensor 210 receives an audiblesignal 212. For example, the audio sensor 210 detects the audible signal50 shown in FIGS. 1A-1D. In some implementations, the audio sensor 210includes a microphone. In various implementations, the system 200 (e.g.,the audio sensor 210 or an audio processing unit) converts the audiblesignal 212 into electronic signal data 214. In various implementations,the electronic signal data 214 is associated with a set of one or moreaudio characteristic values 216 (e.g., the first audio characteristicvalue 52 a shown in FIG. 1A or the second audio characteristic value 52b shown in FIG. 1D). For example, in some implementations, theelectronic signal data 214 is associated with an amplitude value 218that indicates an amplitude of the audible signal 212. Examples of theaudio characteristic values 216 include a frequency value, locationcoordinate values indicating where the audible signal 212 is originatingfrom, and a source type value indicating a source of the audible signal212. In various implementations, the audio sensor 210 provides theelectronic signal data 214 to the data obtainer 220.

In various implementations, the data obtainer 220 obtains environmentaldata 222 (e.g., the environmental data 60 shown in FIG. 1B). In someimplementations, the environmental data 222 includes depth data 224 thatis captured by a depth sensor (e.g., a depth camera). In someimplementations, the environmental data 222 includes image data 226(e.g., a set of one or more images) that is captured by an image sensor(e.g., a camera). In some implementations, the environmental data 222includes noise coefficients 228 (e.g., STC ratings or NRCs for variousmaterials and/or objects in a physical environment) that indicate soundabsorption or reflectiveness properties of objects in the physicalenvironment.

In various implementations, the sound propagation determiner 230determines (e.g., estimates) a distance 234 at which the audible signal212 is audible based on the audio characteristic values(s) 216associated with the electronic signal data 214 and the environmentaldata 222. In some implementations, the distance 234 is measured from asource of the audible signal 212. For example, in FIGS. 1A-1D, thedistance 234 is measured from the user 30 because the user 30 is thesource of the audible signal 50. However, in FIG. 1E, the distance ismeasured from the television represented by the XR television 152because the television is the source of the audible signal.

In various implementations, the sound propagation determiner 230utilizes a sound propagation model 232 to determine the distance 234. Insome implementations, the system 200 (e.g., the sound propagationdeterminer 230) synthesizes the sound propagation model 232 based on theenvironmental data 222. For example, in some implementations, the soundpropagation determiner 230 identifies locations and physical dimensionsof obstructions in the physical environment based on the depth data 224and/or the image data 226. In such implementations, the soundpropagation model 232 models how an audible signal is reflected and/orabsorbed by the obstructions in the physical environment. In someimplementations, the sound propagation model 232 utilizes the noisecoefficients 228 to determine the propagation trajectory of an audiblesignal through the physical environment.

In various implementations, the content presenter 240 obtains a valuerepresenting the distance 234 from the sound propagation determiner 230,and displays an indicator 242 to indicate the distance 234. For example,the content presenter 240 displays the indicator 70 shown in FIG. 1B,the indicator 70 a shown in FIG. 1C, the indicator 70 b shown in FIG. 1Dand/or the bubble 170 shown in FIG. 1E. In some implementations, thecontent presenter 240 displays an XR environment (e.g., the XRenvironment 110 shown in FIGS. 1C and 1D, and/or the XR environment 150shown in FIG. 1E), and the content presenter 240 displays the indicator242 within the XR environment.

FIG. 3 is a flowchart representation of a method 300 of displaying avisual indication of audibility for an audible signal. In variousimplementations, the method 300 is performed by a device with a display(e.g., an optical see-through display, for example, the opticalsee-through display 420 shown in FIG. 4B), an audio sensor (e.g., amicrophone), a non-transitory memory, and one or more processors coupledwith the display, the audio sensor and the non-transitory memory (e.g.,the electronic device 20 shown in FIGS. 1A-1E, and/or the system 200shown in FIG. 2 ). In some implementations, the method 300 is performedby processing logic, including hardware, firmware, software, or acombination thereof. In some implementations, the method 300 isperformed by a processor executing code stored in a non-transitorycomputer-readable medium (e.g., a memory).

As represented by block 310, in some implementations, the method 300includes receiving, via the audio sensor, an audible signal andconverting the audible signal to electronic signal data. For example, asshown in FIG. 2 , the audio sensor 210 receives the audible signal 212and converts the audible signal 212 to the electronic signal data 214.The electronic signal data can be stored in the non-transitory memory.

As represented by block 310 a, in some implementations, the audiblesignal corresponds to an utterance by a user of the device. In someimplementations, the audible signal corresponds to speech (e.g., spokenwords or phrases) from the user of the device. For example, as shown inFIG. 1A, the audible signal 50 originates from the user 30.

As represented by block 310 b, in some implementations, the audiblesignal is generated by the device and is output via a speaker of thedevice. For example, in some implementations, the electronic device 20plays music or a video with sound.

As represented by block 310 c, in some implementations, the audiblesignal is being output by another device. In some implementations, theaudible signal is being output by a television, a speaker, avoice-activated device, etc. For example, as shown in FIG. 1E, thetelevision represented by the XR television 152 outputs the audiblesignal.

As represented by block 320, in some implementations, the method 300includes obtaining environmental data that indicates audio responsecharacteristics of a physical environment in which the device islocated. For example, as shown in FIG. 1B, the electronic device 20obtains the environmental data 60 that indicates audio responsecharacteristics of the physical environment 10. As another example, asshown in FIG. 2 , the data obtainer 220 obtains the environmental data222.

As represented by block 320 a, in some implementations, obtaining theenvironmental data includes receiving the environmental data via anenvironmental sensor. In some implementations, the environmental sensorincludes a depth sensor (e.g., a depth camera). In some implementations,the environmental sensor includes an image sensor (e.g., a camera). Insome implementations, the environmental sensor includes the audio sensorthat is used to receive the audible sensor.

As represented by block 320 b, in some implementations, obtaining theenvironmental data includes capturing an image of the physicalenvironment, and determining the audio response characteristics of thephysical environment by performing a scene analysis of the physicalenvironment. For example, as shown in FIG. 2 , in some implementations,the environmental data 222 includes the image data 226 (e.g., a set ofone or more images capturing the physical environment).

As represented by block 320 c, in some implementations, obtaining theenvironmental data includes capturing depth data that indicateslocations of physical obstructions in the physical environment. Forexample, as shown in FIG. 2 , in some implementations, the environmentaldata 222 includes depth data 224. In some implementations, the physicalobstructions include surfaces such as a floor, a ceiling and walls. Insome implementations, the physical obstructions include furniture.

In some implementations, the method 300 includes determining the audioresponse characteristics of the physical environment based on materialcomposition and physical dimensions of physical obstructions in thephysical environment. In some implementations, the method 300 includesdetermining whether the physical obstructions reflect sound or absorbsound and to what extent.

In some implementations, the method 300 includes obtaining soundabsorption coefficients or sound reflection coefficients of materialsfrom which the physical obstructions are made. For example, as shown inFIG. 2 , in some implementations, the environmental data 222 includesnoise coefficients 228 (e.g., STC ratings or NRC values for variousmaterials used in the physical environment).

As represented by block 320 d, in some implementations, the method 300includes receiving, via the audio sensor, another audible signal thatrepresents an echo of the audible signal, and determining the audioresponse characteristics based on the echo. For example, in someimplementations, the method 300 includes determining the audio responsecharacteristics of the physical environment based on an amplitude of theecho. In some implementations, the method 300 includes determining theaudio response characteristics of the physical environment based on anamount of time that passes between detecting the audible signal anddetecting an echo of the audible signal.

As represented by block 330, in some implementations, the method 300includes displaying, on the display, an indicator that indicates adistance from a source of the audible signal at which the audible signalis audible. In some implementations, the distance is based on anamplitude associated with the electronic signal data and the audioresponse characteristics of the physical environment. In someimplementations, the distance represents a distance at which the audiblesignal is intelligible to a person. In some implementations, thedistance represents a distance at which the audible signal is detectableby a device such as a voice-activated virtual assistant device.

In various implementations, displaying the indicator enhances afunctionality of the electronic device by providing the user a visualindication of whether an amplitude of the audible signal needs to bedecreased in order to prevent the audible signal from being audible in aparticular portion of the physical environment. In variousimplementations, displaying the indicator enhances a functionality ofthe electronic device by providing the user a visual indication ofwhether an amplitude of the audible signal needs to be increased inorder to allow the audible signal to be audible in a particular portionof the physical environment.

As represented by block 330 a, in some implementations, the method 300includes generating, based on the audio response characteristics of thephysical environment, a sound propagation model that models propagationof audible signals in the physical environment. For example, as shown inFIG. 2 , the sound propagation determiner 230 generates a soundpropagation model 232 that models the propagation of the audible signal212 in a physical environment. In some implementations, the method 300includes providing an audio characteristic value (e.g., an amplitudevalue) associated with the electronic signal data as an input to thesound propagation model, and receiving the distance as an output of thesound propagation model.

As represented by block 330 b, in some implementations, the distanceindicates how far the audible signal is propagating from the sourcebased on the amplitude of the audible signal and the audio responsecharacteristics of the physical environment. For example, as shown inFIG. 1C, the indicator 70 a indicates that the audible signal 50 isaudible at a location that corresponds to the indicator 70 a.

As represented by block 330 c, in some implementations, the distanceindicates how far from the source the audible signal can be heard by aperson. In some implementations, the distance indicates how far theaudible signal is intelligible by a person. For example, the distanceindicates how far from the source of the audible signal the audiblesignal can be properly heard and comprehended by a person with averagehearing abilities.

In some implementations, the indicator indicates whether or not theaudible signal can be heard by a person located at the distance. Forexample, as shown in FIG. 1C, the indicator 70 a indicates that theaudible signal can be heard by the person 40 because the XR person 140is on the concave side of the indicator 70 a (e.g., the XR person isinside a bubble represented by the indicator 70 a). However, theindicator 70 b shown in FIG. 1D indicates that the audible signal 50 isinaudible to the person 40 because the XR person 140 is on the convexside of the indicator 70 b (e.g., the XR person is outside a bubblerepresented by the indicator 70 b).

As represented by block 330 d, in some implementations, the distanceindicates how far from the device the audible signal can be detected bya voice-activated device that can perform an operation in response todetecting the audible signal. In some implementations, the indicatorindicates whether or not the audible signal can be detected by avoice-activated device located at the distance. As such, if the user istrying to activate the voice-activated device and the voice-activateddevice is not activating, the indicator indicates whether the user needsto speak louder in order to activate the voice-activated device.

As represented by block 330 e, in some implementations, the indicator issphere-shaped. In some implementations, the indicator is a bubble. Forexample, as shown in FIG. 1E, the electronic device 20 displays thebubble 170 to indicate how far an audible signal generated by thetelevision is propagating in the XR environment 150.

In some implementations, the method 300 includes receiving, from asecond device that includes a second audio sensor, an indicationindicating whether the second audio sensor is receiving the audiblesignal. In some implementations, the method 300 further includesdetermining the distance based on the indication receiving from thesecond device.

In some implementations, the second device is located at the distanceindicated by the indicator. In such implementations, if the seconddevice indicates that the second device is able to detect the audiblesignal, displaying the indicator indicates that the audible signal isreaching the second device.

In some implementations, the second device is located at a seconddistance that is smaller than the distance indicated by the indicator.In such implementations, the device determines (e.g., estimates) how farthe audible signal is propagating based on a strength (e.g., anamplitude) of the audible signal detected by the second device.

In some implementations, the distance is further based on a proximity ofthe device to the source of the audible signal. As such, in someimplementations, the method 300 includes determining a position of thesource of the audible signal relative to the device, and determining thedistance at which the audible signal, generated by the source, isaudible based on the relative position of the source. In someimplementations, the amplitude associated with the electronic signaldata is a function of a distance between the device and the source. Forexample, in some implementations, the amplitude is inverselyproportional to the distance between the device and the source (e.g.,the amplitude decreases as the distance between the device and thesource increases, and the amplitude increases as the distance betweenthe device and the source decreases).

FIG. 4A is a block diagram of a device 400 in accordance with someimplementations. In some implementations, the device 400 implements theelectronic device 20 shown in FIGS. 1A-1E, and/or the system 200 shownin FIG. 2 . While certain specific features are illustrated, those ofordinary skill in the art will appreciate from the present disclosurethat various other features have not been illustrated for the sake ofbrevity, and so as not to obscure more pertinent aspects of theimplementations disclosed herein. To that end, as a non-limitingexample, in some implementations the device 400 includes one or moreprocessing units (CPUs) 401, a network interface 402, a programminginterface 403, a memory 404, one or more input/output (I/O) devices 410,and one or more communication buses 405 for interconnecting these andvarious other components.

In some implementations, the network interface 402 is provided to, amongother uses, establish and maintain a metadata tunnel between a cloudhosted network management system and at least one private networkincluding one or more compliant devices. In some implementations, theone or more communication buses 405 include circuitry that interconnectsand controls communications between system components. The memory 404includes high-speed random access memory, such as DRAM, SRAM, DDR RAM orother random access solid state memory devices, and may includenon-volatile memory, such as one or more magnetic disk storage devices,optical disk storage devices, flash memory devices, or othernon-volatile solid state storage devices. The memory 404 optionallyincludes one or more storage devices remotely located from the one ormore CPUs 401. The memory 404 comprises a non-transitory computerreadable storage medium.

In some implementations, the memory 404 or the non-transitory computerreadable storage medium of the memory 404 stores the following programs,modules and data structures, or a subset thereof including an optionaloperating system 406, the data obtainer 220, the sound propagationdeterminer 230, and the content presenter 240. In variousimplementations, the device 400 performs the method 300 shown in FIG. 3.

In some implementations, the data obtainer 220 obtains environmentaldata that indicates audio response characteristics of a physicalenvironment. In some implementations, the data obtainer 220 performs theoperation(s) represented by block 320 in FIG. 3 . To that end, the dataobtainer 220 includes instructions 220 a, and heuristics and metadata220 b.

In some implementations, the sound propagation determiner 230 determinesa distance from the device at which the audible signal is audible. Tothat end, the sound propagation determiner 230 includes instructions 230a, and heuristics and metadata 230 b.

In some implementations, the content presenter 240 presents an indicatorthat indicates the distance determined by the sound propagationdeterminer 230. In some implementations, the sound propagationdeterminer 230 and the content presenter 240 collectively perform theoperation(s) represented by block 330 shown in FIG. 3 . To that end, thecontent presenter 240 includes instructions 240 a, and heuristics andmetadata 240 b.

In some implementations, the one or more I/O devices 410 include anaudio sensor (e.g., a microphone) for receiving an audible signal (e.g.,the audible signal 50 shown in FIGS. 1A-1D, or the audible signal 212shown in FIG. 2 ). In some implementations, the one or more I/O devices410 include an image sensor (e.g., a camera) to capture the image data226 shown in FIG. 2 . In some implementations, the one or more I/Odevices 410 include a depth sensor (e.g., a depth camera) to capture thedepth data 224 shown in FIG. 2 . In some implementations, the one ormore I/O devices 410 include a display (e.g., the display 22 shown inFIG. 1B) for displaying an indicator (e.g., the indicator 70 shown inFIG. 1B, the indicator 70 a shown in 1C, the indicator 70 b shown in 1D,the bubble 170 shown in FIG. 1E, or the indicator 242 shown in FIG. 2 ).In some implementations, the one or more I/O devices 410 include aspeaker for outputting an audible signal (e.g., the audible signal 50shown in FIGS. 1A-1D, or the audible signal 212 shown in FIG. 2 ).

In various implementations, the one or more I/O devices 410 include avideo pass-through display which displays at least a portion of aphysical environment surrounding the device 400 as an image captured bya scene camera. In various implementations, the one or more I/O devices410 include an optical see-through display which is at least partiallytransparent and passes light emitted by or reflected off the physicalenvironment.

FIG. 4B illustrates a blow-up view of an optical see-through display 420in accordance with some implementations. In various implementations, theoptical see-through display 420 includes a selectively occlusive layer450 that includes a number of pixel elements that, when activated, blocklight from passing through the optical see-through display 420. Thus,through appropriate addressing of the selectively occlusive layer 450,the optical see-through display 420 can render a black region 451 or agray region 452. In various implementations, the optical see-throughdisplay 420 includes a globally dimmable layer 460 that, according to acontrollable dimming level, dims light passing through the opticalsee-through display 420. In various implementations, the globallydimmable layer 460 includes one or more of a photochromic element,electrochromic element, an SPD (suspended-particle device) element, GHLC(guest-host liquid crystal) element, or PDLC (polymer-dispersedliquid-crystal) element. In various implementations, the opticalsee-through display 420 includes a light addition layer 470 thatincludes a number of pixel elements that, when activated, emit lighttowards the user. Thus, through appropriate addressing of the lightaddition layer 470, the optical see-through display 420 can render awhite (or colored) virtual object 471. In various implementations, theoptical see-through display 420 does not include each of the layers 450,460, 470. In particular, in various implementations, the opticalsee-through display 420 does not include the selectively occlusive layer450 and/or the globally dimmable layer 460. In various implementations,the optical see-through display 420 does not include the light additionlayer 470 and/or the globally dimmable layer 460. In variousimplementations, the optical see-through display 420 does not includethe selectively occlusive layer 450 and/or the light addition layer 470.

Various processes defined herein consider the option of obtaining andutilizing a user's personal information. For example, such personalinformation may be utilized in order to provide an improved privacyscreen on an electronic device. However, to the extent such personalinformation is collected, such information should be obtained with theuser's informed consent. As described herein, the user should haveknowledge of and control over the use of their personal information.

Personal information will be utilized by appropriate parties only forlegitimate and reasonable purposes. Those parties utilizing suchinformation will adhere to privacy policies and practices that are atleast in accordance with appropriate laws and regulations. In addition,such policies are to be well-established, user-accessible, andrecognized as in compliance with or above governmental/industrystandards. Moreover, these parties will not distribute, sell, orotherwise share such information outside of any reasonable andlegitimate purposes.

Users may, however, limit the degree to which such parties may access orotherwise obtain personal information. For instance, settings or otherpreferences may be adjusted such that users can decide whether theirpersonal information can be accessed by various entities. Furthermore,while some features defined herein are described in the context of usingpersonal information, various aspects of these features can beimplemented without the need to use such information. As an example, ifuser preferences, account names, and/or location history are gathered,this information can be obscured or otherwise generalized such that theinformation does not identify the respective user.

While various aspects of implementations within the scope of theappended claims are described above, it should be apparent that thevarious features of implementations described above may be embodied in awide variety of forms and that any specific structure and/or functiondescribed above is merely illustrative. Based on the present disclosureone skilled in the art should appreciate that an aspect described hereinmay be implemented independently of any other aspects and that two ormore of these aspects may be combined in various ways. For example, anapparatus may be implemented and/or a method may be practiced using anynumber of the aspects set forth herein. In addition, such an apparatusmay be implemented and/or such a method may be practiced using otherstructure and/or functionality in addition to or other than one or moreof the aspects set forth herein.

It will also be understood that, although the terms “first”, “second”,etc. may be used herein to describe various elements, these elementsshould not be limited by these terms. These terms are only used todistinguish one element from another. For example, a first node could betermed a second node, and, similarly, a second node could be termed afirst node, which changing the meaning of the description, so long asall occurrences of the “first node” are renamed consistently and alloccurrences of the “second node” are renamed consistently. The firstnode and the second node are both nodes, but they are not the same node.

The terminology used herein is for the purpose of describing particularimplementations only and is not intended to be limiting of the claims.As used in the description of the implementations and the appendedclaims, the singular forms “a”, “an”, and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. It will also be understood that the term “and/or” as usedherein refers to and encompasses any and all possible combinations ofone or more of the associated listed items. It will be furtherunderstood that the terms “comprises” and/or “comprising”, when used inthis specification, specify the presence of stated features, integers,steps, operations, elements, and/or components, but do not preclude thepresence or addition of one or more other features, integers, steps,operations, elements, components, and/or groups thereof.

As used herein, the term “if” may be construed to mean “when” or “upon”or “in response to determining” or “in accordance with a determination”or “in response to detecting”, that a stated condition precedent istrue, depending on the context. Similarly, the phrase “if it isdetermined [that a stated condition precedent is true]” or “if [a statedcondition precedent is true]” or “when [a stated condition precedent istrue]” may be construed to mean “upon determining” or “in response todetermining” or “in accordance with a determination” or “upon detecting”or “in response to detecting” that the stated condition precedent istrue, depending on the context.

What is claimed is:
 1. A method comprising: at a device including adisplay, an audio sensor, a processor and a non-transitory memory:receiving, via the audio sensor, an audible signal and converting theaudible signal to electronic signal data; obtaining environmental datathat indicates audio response characteristics of a physical environmentin which the device is located; and displaying, on the display, anindicator that indicates a distance from a source of the audible signalat which the audible signal is audible, wherein the distance is based onan amplitude associated with the electronic signal data and the audioresponse characteristics of the physical environment.
 2. The method ofclaim 1, wherein obtaining the environmental data comprises receivingthe environmental data via an environmental sensor.
 3. The method ofclaim 1, wherein obtaining the environmental data comprises capturing animage of the physical environment and determining the audio responsecharacteristics of the physical environment by performing a sceneanalysis of the physical environment.
 4. The method of claim 1, whereinobtaining the environmental data comprises capturing depth data thatindicates locations of physical obstructions in the physicalenvironment.
 5. The method of claim 4, further comprising determiningthe audio response characteristics of the physical environment based onmaterial composition and physical dimensions of the physicalobstructions in the physical environment.
 6. The method of claim 5,further comprising obtaining sound absorption coefficients or soundreflection coefficients of materials from which the physicalobstructions are made.
 7. The method of claim 1, further comprising:receiving, via the audio sensor, another audible signal that representsan echo of the audible signal; and determining the audio responsecharacteristics based on the echo.
 8. The method of claim 1, furthercomprising generating, based on the audio response characteristics ofthe physical environment, a sound propagation model that modelspropagation of audible signals in the physical environment.
 9. Themethod of claim 8, further comprising providing an audio characteristicvalue associated with the electronic signal data as an input to thesound propagation model, and receiving the distance as an output of thesound propagation model.
 10. The method of claim 1, wherein the distanceindicates how far the audible signal is propagating from the sourcebased on the amplitude and the audio response characteristics.
 11. Themethod of claim 1, wherein the distance indicates how far from thesource the audible signal can be heard by a person.
 12. The method ofclaim 1, wherein the indicator indicates whether or not the audiblesignal can be heard by a person located at the distance.
 13. The methodof claim 1, wherein the distance indicates how far from the source theaudible signal can be detected by a voice-activated device that canperform an operation in response to detecting the audible signal. 14.The method of claim 1, wherein the indicator indicates whether or notthe audible signal can be detected by a voice-activated device locatedat the distance.
 15. The method of claim 1, wherein the indicator issphere-shaped.
 16. The method of claim 1, wherein the audible signalcorresponds to an utterance by a user of the device.
 17. The method ofclaim 1, wherein the audible signal is generated by the device and isoutput via a speaker of the device.
 18. The method of claim 1, whereinthe audible signal is being output by another device.
 19. Anon-transitory memory storing one or more programs, which, when executedby one or more processors of a device including an audio sensor and adisplay, cause the device to: receive, via the audio sensor, an audiblesignal and converting the audible signal to electronic signal data;obtain environmental data that indicates audio response characteristicsof a physical environment in which the device is located; and display,on the display, an indicator that indicates a distance from a source ofthe audible signal at which the audible signal is audible, wherein thedistance is based on an amplitude associated with the electronic signaldata and the audio response characteristics of the physical environment.20. A device comprising: an audio sensor; one or more processors; anon-transitory memory; one or more displays; and one or more programsstored in the non-transitory memory, which, when executed by the one ormore processors, cause the device to: receive, via the audio sensor, anaudible signal and convert the audible signal to electronic signal data;obtain environmental data that indicates audio response characteristicsof a physical environment in which the device is located; and display,on the one or more displays, an indicator that indicates a distance froma source of the audible signal at which the audible signal is audible,wherein the distance is based on an amplitude associated with theelectronic signal data and the audio response characteristics of thephysical environment.
 21. The non-transitory memory of claim 19, whereinobtaining the environmental data comprises capturing an image of thephysical environment and determining the audio response characteristicsof the physical environment by performing a scene analysis of thephysical environment.
 22. The non-transitory memory of claim 19, whereinobtaining the environmental data comprises capturing depth data thatindicates locations of physical obstructions in the physicalenvironment.
 23. The non-transitory memory of claim 19, wherein the oneor more programs further cause the device to: receive, via the audiosensor, another audible signal that represents an echo of the audiblesignal; and determine the audio response characteristics based on theecho.
 24. The device of claim 20, wherein the one or more programsfurther cause the device to generate, based on the audio responsecharacteristics of the physical environment, a sound propagation modelthat models propagation of audible signals in the physical environment.25. The device of claim 20, wherein the distance indicates how far fromthe source the audible signal can be heard by a person.